Stream data processing has been attracting attention as a method for quickly responding to the need for analyzing a large amount of continuously generated data in real time, such as the analysis of automatic stock trading, advanced traffic information processing, and sensor information obtained at multiple locations. The stream data processing is a general purpose middleware technology that can be applied to real-time processing of data with different formats. This allows reflecting the real-world data in business in real time, while responding to rapid changes in the business environment that are too fast to catch up to by establishing a system for each case. The principle of the stream data processing and the implementation method thereof are disclosed in Non-patent Literature 1.
As described above, the stream data processing is real time processing of a large amount of data, so that the output data of processing results are continuously generated. Thus, it is desirable that the time required for the recovery from the occurrence of a failure should be reduced as much as possible. At this time, the running state of the restored server is the initial state, so that it is necessary to provide running state reproduction in which the running state before the occurrence of a failure is also reproduced in the restored server.
The first method of running state reproduction is the upstream backup method disclosed in Non-patent literature 2. In the upstream backup method, the input data is backed up during normal operation. Then, upon recovery the backup data is re-executed by a standby server to catch up to the running state of the currently used server. The longer the processing time, the larger the storage amount of the disk and memory. However, it can be assumed that the storage amount is kept within a certain range due to the following reasons.
The stream data processing can use window operations to cut out the latest part of the data series. The definition of the window operation is disclosed in Non-patent literature 3. For example, the aggregate function is applied to the data that is cut out by a window operation for the duration of one minute to calculate the median, resulting in the operation of the calculation of the moving average for one minute. In this example, when the data is allowed to flow for one minute, the data in the window is renewed. This means that when recovery is started from the initial state, the running state returns to the running state before failure by processing the data for the last one minute. As described above, in the upstream backup method, it can be assumed that the amount of storage for backup is within a certain range based on the assumption that the range of data to be held moves to the future with the progression of the process.
The second method of running state reproduction is as follows. First, the running state is made static by periodically interrupting the running server. Then, static running state is stored as a replication (snapshot). In this way, when a failure occurs and restoration takes place, the running state is reproduced from the stored snapshot. The method of making the running state static and storing the snapshot is widely used in the database and transaction systems. The reproduction method using the static approach in an in-memory database is disclosed in Patent literature 1.